Repository navigation
Conversation
6763b14 to
59b0397
Compare
6a3ea63 to
d56813b
Compare
d56813b to
de4b325
Compare
|
Considering the importance and complexity of the PR, AI was also used to review the behaviour. Issues1. Hook ordering puts the idempotency check after the audit insert → duplicate rows, then hard failure on the 3rd callOrders in the single
Trace of requests with key
We might want to consider use Two independent fixes are needed:
Note this is also a regression relative to 2. No atomic key claim → the idempotency guarantee doesn't hold under concurrency
This needs a unique index on We should eagerly store incoming command + idempotency and fail-fast if such entries already exists: replay till result is gathered / timeout. 3. In-flight and failed commands are not detected
The store already exposes
4.
|
|
@adamsaghy the order of the hooks was actually intentional... have to look at the notes concerning potential failure scenarios, but as for the order you could look at this also as a feature... if we do as suggested then you'll have no trace a duplicate attempt ever happened (could be malicious)... if that works for you then good for me. Please confirm before I start changing. |
|
Concerning the scenario 7: |
|
Concerning scenario 6: I'm not entirely sure if this is a false flag, but async and disruptor mode are disabled and unusable anyway. For now this is out of scope. |
|
Concerning minor issue "The check hook needs @ConditionalOnBean(CommandStore.class)": this is the intended behavior. A store should always be available. If we don't then it should not be possible to start. And that's the case here. |
|
Concerning rest of minor issues: most of them fixed. |
|
Concerning scenario 5: check added. |
|
Concerning scenario 4: Introduced |
|
Concerning scenario 3: Added a new query that checks for existence of command by idempotency without relying on response object. Should be fixed. |
|
Concerning scenario 2: Added unique constraint and made sure that any duplicate keys get removed from storage to avoid conflicts when the Liquibase script is executed. |
de4b325 to
efc3798
Compare
|
Concerning scenario 1: changed hook order, idempotency now before audit, but the core hooks are still running first; this is intentional because if something fails I want to have some context (e.g. IP address); introduced a |
efc3798 to
de98fbf
Compare
|
This pull request seems to be stale. Are you still planning to work on it? We will automatically close it in 30 days. |
|
@vidakovic Please rebase onto the latest |
galovics
left a comment
There was a problem hiding this comment.
Hi @vidakovic, thanks for working through Adam's list, the CommandContext refactor makes the hook API a lot cleaner. But I think the idempotency part still doesn't work end to end. As far as I can tell a legit replay gets null back, and two concurrent requests with the same key still both run the handler. On top of that the MySQL integration tests are red on every shard, and nothing in the PR tests the new module. Details inline.
Also, the branch is from mid-August. Since then develop moved to Spring Boot 4.1 and JDK 25, so please rebase as Adam asked. CI needs a fresh run anyway.
fineract-command-jdbc/src/main/java/org/apache/fineract/command/jdbc/store/JdbcCommandStore.java:75
public boolean checkRequestInstanceByKey(String key, Class<?> clazz) {
return repository.findOneByIdempotencyKey(key).map(CommandEntity::getResponse)
.map(json -> clazz.getCanonicalName().equals(json.get(COMMAND_JSON_CLASS_ATTRIBUTE).asText())).orElse(false);Wait, this compares the request payload class against the
@classof the stored response. Those are basically never the same type (DummyRequestvsDummyResponse), so this returns false for a proper replay too. Shouldn't this beCommandEntity::getRequest?
fineract-command-idempotency/src/main/java/org/apache/fineract/command/idempotency/hook/IdempotencyCheckCommandHook.java:54
ctx.setSkipExecution(true);
ctx.setSkipHooks(true);
var isSameRequestClass = store.checkRequestInstanceByKey(command.getIdempotencyKey(), command.getPayload().getClass());
...
// TODO: else throw an exception?Related to the above. Whenever the key exists, we skip execution and hand back whatever is in
ctx.getResponse(). With the class check broken that's alwaysnull. It's alsonullwhen the first request is stillUNDER_PROCESSING, ended inERROR, or used the same key on a different endpoint. So the client gets an empty success response and can't tell that nothing happened. I'd resolve the TODO in this PR rather than later. Key reuse with a different request should be a 4xx, in-flight should be a 409, and forERRORwe need to decide explicitly whether a retry is allowed. Returningnullsilently is the worst option for financial commands. Thoughts?
fineract-command-jdbc/src/main/java/org/apache/fineract/command/jdbc/store/JdbcCommandStore.java:116
void fallback(CommandContext<?, ?> ctx, Throwable t) throws Exception {
...
log.warn("Command store fallback idempotencyKey={}, ...", ..., t);I'm not convinced the new unique constraint actually prevents double execution. Two concurrent requests with key K both pass
existsByKeyat order 21. The second one then hits the unique violation inAuditCommandHookBefore->store(). That exception isn't inretryExceptions, so it goes straight into this fallback, which logs and swallows it, and the handler runs a second time anyway. For idempotency the claim insert has to fail the request (409 or replay), not end up in the dead-letter path. Probably the duplicate-key case needs to be caught explicitly and rethrown from the hook rather than routed through@Retry.
fineract-command-jdbc/src/main/resources/db/changelog/tenant/module/command/parts/0003_idempotency_key_unique.xml:26
<delete tableName="m_command">
<where>
idempotency_key IN (
SELECT c1.idempotency_key
FROM m_command c1Two things here. First, MySQL doesn't allow a DELETE whose subquery reads the same table (error 1093, "You can't specify target table ... for update in FROM clause"). MariaDB and PG accept it, which matches the CI picture exactly: all 15 MySQL shards fail because the app never comes up, while MariaDB and PG are green. Wrapping the subquery in a derived table (
SELECT ... FROM (SELECT ...) t) or doing it withdbms-specific SQL should fix that.Second, this deletes every row whose key has a duplicate, including the oldest one. On an audit table I'd rather keep the first row and only delete the later ones (
id IN (SELECT c1.id ... WHERE c1.id > c2.id)). Even better would be to null out the key on the duplicates instead of deleting audit history. What do you think?
fineract-command-audit/src/main/java/org/apache/fineract/command/audit/check/AuditCommandPropertyCheck.java:19
package org.apache.fineract.command.audit.check;Does this ever run?
AuditCommandPropertyCheckAutoConfigurationscansCOMMAND_AUDIT_HOOK_CHECK_PACKAGE, which isorg.apache.fineract.command.audit.hook.check, but the class lives inorg.apache.fineract.command.audit.check. Also,containsPropertyonly checks that the property is present, sohook-pre-enabled=falsewould log "enabled". I think you wantenvironment.getProperty(..., Boolean.class, true).
fineract-command/src/test/java/org/apache/fineract/command/CommandSampleApiTest.java:96
- final var idempotencyKey = UUID.randomUUID().toString();
+ // final var idempotencyKey = UUID.randomUUID().toString();Why commented? This was the one test sending an idempotency key, and it's now gone. The new
fineract-command-idempotencymodule has no tests at all. For a feature whose whole point is "never execute a financial command twice", I'd really like to see at least: same key + same request replays the original response, same key + different request is rejected, a retry while the first call is in flight, a retry after anERROR, and two concurrent requests with the same key on a real DB (Testcontainers, there's already a base test infineract-command-test). The first case would have caught thegetResponsemixup above.
fineract-command-disruptor/src/main/java/org/apache/fineract/command/disruptor/implementation/DisruptorCommandDispatcher.java:130
} finally {
event.setFuture(new CompletableFuture<>());
}Not blocking since disruptor is off anyway, but this trades the old stale-future bug for a race.
dispatch()readsprocessedEvent.getFuture()(line 61) afterringBuffer.publish(line 84). If the handler finishes first, the caller picks up the fresh future, which belongs to the next command in that slot or never completes. I'd create the future innext()before publishing and return it from there.
Description
Implement idempotency feature for new command processing.
Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!
Your assigned reviewer(s) will follow our guidelines for code reviews.